Refactor tinygrid/ercot.py into modular subpackage#2
Conversation
This refactoring breaks the monolithic ercot.py (3680 lines) into a well-organized subpackage with clear separation of concerns: ## New Module Structure - `tinygrid/ercot/__init__.py` - Main ERCOT class combining mixins - `tinygrid/ercot/client.py` - ERCOTBase with auth, retry, pagination - `tinygrid/ercot/endpoints.py` - Low-level pyercot wrappers (~100 methods) - `tinygrid/ercot/api.py` - High-level unified API (get_spp, get_lmp, etc.) - `tinygrid/ercot/archive.py` - Historical archive access (moved from historical/) - `tinygrid/ercot/dashboard.py` - Public dashboard methods (no auth required) - `tinygrid/ercot/documents.py` - MIS document fetching for yearly data - `tinygrid/ercot/transforms.py` - Data filtering and transformation utilities ## New Features - Dashboard methods: get_status(), get_fuel_mix(), get_energy_storage_resources() - MIS document fetching: _get_documents(), read_doc(), REPORT_TYPE_IDS - Historical yearly methods: get_rtm_spp_historical(year), get_dam_spp_historical(year) - Renamed decorator: @support_date_range → @with_date_range (backward-compatible) ## Architecture Improvements - Mixin-based design for composable functionality - Standalone transform functions separate data manipulation from API logic - Cleaner import structure with proper re-exports for backward compatibility All 505 tests pass.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| logger.error(f"Failed to parse document {doc.doc_id}: {e}") | ||
| return pd.DataFrame() | ||
|
|
||
| return pd.DataFrame() |
There was a problem hiding this comment.
Zip files without CSVs silently return empty DataFrame
In read_doc, when a URL contains .zip but the zip file contains no CSV files, the control flow falls through to line 253 and returns an empty DataFrame without any logging or warning. All other code paths either return successfully or log an error before returning. This silent failure can make debugging difficult when users expect data but receive an empty DataFrame without explanation.
| products = response.get("products", []) | ||
| if not products: | ||
| return pd.DataFrame() | ||
| return pd.DataFrame(products) |
There was a problem hiding this comment.
Products extraction fails if API returns HAL format
The simplified _products_to_dataframe method only extracts products from response.get("products", []). The old implementation explicitly handled HAL format with _embedded.products, nested additional_properties._embedded.products, and list responses. If pyercot's API returns products in HAL format (which the old code explicitly supported), get_list_for_products() will silently return an empty DataFrame even when products exist, because _embedded nesting is not unwrapped by _extract_response_data.
- Fix document download URL parameter (docid -> doclookupId) - Fix ZIP file handling for Excel and CSV extraction - Add openpyxl dependency for Excel file reading - Update get_settlement_point_mapping to return dict of all 5 tables - Update dashboard methods to be placeholder stubs with warnings - Add comprehensive tests for documents module (25 tests) - Add tests for dashboard module (22 tests) - Update READMEs and notebook to reflect working functionality - Coverage now at 91% (552 tests)
Co-authored-by: kvkenyon <kvkenyon@utexas.edu>
|
Cursor Agent can help with this pull request. Just |
|
@cursoragent Seems like our codecov is too low for this patch 84.81% and we need 91.44%. Let's improve the coverage. |
|
I have improved the code coverage by adding comprehensive tests for the new modular subpackage structure and fixed a bug in the retry logic. Summary of Changes
Coverage Improvements
These changes should bring the patch coverage significantly above the 91.44% target. Learn more about Cursor Agents |
Co-authored-by: kvkenyon <kvkenyon@utexas.edu>
This commit refactors the tests for the ERCOT API and client. It adds new tests, improves existing ones, and ensures better coverage of edge cases and error handling. Co-authored-by: kvkenyon <kvkenyon@utexas.edu>
3 participants
This refactoring breaks the monolithic ercot.py (3680 lines) into a well-organized subpackage with clear separation of concerns:
New Module Structure
tinygrid/ercot/__init__.py- Main ERCOT class combining mixinstinygrid/ercot/client.py- ERCOTBase with auth, retry, paginationtinygrid/ercot/endpoints.py- Low-level pyercot wrappers (~100 methods)tinygrid/ercot/api.py- High-level unified API (get_spp, get_lmp, etc.)tinygrid/ercot/archive.py- Historical archive access (moved from historical/)tinygrid/ercot/dashboard.py- Public dashboard methods (no auth required)tinygrid/ercot/documents.py- MIS document fetching for yearly datatinygrid/ercot/transforms.py- Data filtering and transformation utilitiesNew Features
Architecture Improvements
Note
Modernizes docs, examples, and tests to match the new modular ERCOT subpackage and adds support deps for MIS document parsing.
examples/ercot_demo.ipynband reference itopenpyxldependency for reading Excel files from MIS archivestinygrid.ercot.endpoints,tinygrid.ercot.client,tinygrid.ercot.archive)tinygrid.ercot.archiveWritten by Cursor Bugbot for commit e46c90e. This will update automatically on new commits. Configure here.